Scaffolded use of a modeling tool to support scientific inquir

Scientists often use models to help develop theories and plan experiments. Computational tools have greatly expanded the sophistication of such models. One such tool, the Oncology Thinking Cap (OncoTCap) provides a comprehensive modeling workbench for cancer researchers. This tool is versatile and can be used to model clinical trials, but its functions and operation may impose excessive cognitive load for the novice user. Moreover, novices may have little knowledge of the kinds of experimental designs used in the cancer domain. There are welldefined sets of designs for clinical trials of new drugs, used at different stages of testing and for different purposes. As Baker and Dunbar (1996) note, the expert scientist often has a schema for the slots of an experimental design that need to be filled. Work on software-realized scaffolding suggests that communicating the design process is one way that learners can begin to construct these schemas. In this paper, we report on the design, use, and evaluation of a special purpose interface, the Phase II Trial Design wizard, to help scaffold students’ design of an experiment to test a new cancer drug. Introduction One way that scientists build theories when they cannot immediately experiment is through the use of models and thought experiments (Nersessian, 1995; Toth, 1996). Computational tools have greatly expanded the sophistication of such models. For novices, with limited experience in a domain, this task may pose excessive cognitive demand. Moreover, the novices may not have constructed an appropriate schema for designing experiments (Baker & Dunbar, 1996). We can help students learn such a schema by communicating to them the important slots that they need to fill in the process of designing their experiments. In addition, allowing students to simulate their experiments and observe the results provides the dynamic feedback needed to enhance their understanding of both the process of experimentation and their knowledge of the domain (Schauble, Glaser, Duschl, Schulze, & John, 1995). In this paper, we describe scaffolded use of a professional modeling tool to help medical students learn to design clinical trials of an anticancer drug. Based on notions of software-realized scaffolding, we have designed a “wizard” to help communicate the process of designing a clinical trial (Hmelo & Guzdial, 1996). This wizard helped set up the trial that the students could then simulate using the Oncology Thinking Cap (OncoTCap), a comprehensive modeling tool for simulating the behavior of heterogeneous population of cancer cells. The slots that need to be filled are highlighted through screen-based forms that provide the needed input for setting up the model in OncoTCap. The process of bringing a new drug to market is complex. A drug must go through several stages of laboratory and clinical testing before it can be available for general use (Simon, 1993) . The first phase of clinical testing involves a small number of patients to identify a safe dose. A phase II trial is conducted to see if there is any biological response to the drug. In this process, the researchers must determine the dosage and scheduling of the drug. The researchers need to specify many of the operational characteristics of the study design, a priori in order to determine how many responses to the drug will make it worthy of further study. A phase III trial compares the new drug against existing treatments to determine whether it is more effective than existing treatments. In this study, we developed scaffolding to help students learn to design Phase II clinical trials, allowed them to simulate running their trials, and then go through an iterative design process to optimize their design. We hypothesized that there were several aspects of clinical trial design that were important, some of which are not salient to novices. The more obvious elements of a Phase II clinical trial design include: • Understanding the background information about the disease being treated and the drug being tested • Criteria for including/ excluding patients • Considering the results of preclinical and phase I testing Specifying the dosage, treatment schedule, and treatment duration Some of the schema elements that should be less obvious to the students include: • Conditional rules such as planning to modify the drug dosage as a result of toxic side effects • Conditions under which the patient will stop receiving the experimental protocols • Operational characteristics needed for determining how many patients will be included in the study, and of that group, how many responses need to be observed to conclude that the drug is worthy of further study. We have implemented the Phase II Clinical Trial Wizard (Trial Wizard) on the foundation of OncoTCap. A description of OncoTCap and the Wizard are provided in the following sections. OncoTCap The Oncology Thinking Cap is a computer-based modeling laboratory for conducting experiments in cancer research and treatment. The core concept underlying OncoTCap is the principle that tumor heterogeneity forms the basis for most of the concepts in cancer biology and treatment (Fuji, Marsh, Cairns, Sidransky, & Gabrielson, 1996). OncoTCap can model the various components in cancer research and treatment such as cell cycle control, apoptotic mechanisms, mutational processes, cell repair mechanisms, cell kinetics, treatment characteristics, treatment resistance, treatment schedules, cell genetic characteristics (Day, Shirey, Ramakrishnan, & Huang, 1998). Using tumor cell heterogeneity, OncoTCap models these processes by specifying the different properties of cancer cells such as their genetic make-up, location and resistance characteristics (Ramakrishnan, Hmelo, Day, Shirey, & Huang, 1998). For the purpose of forming the foundation for the Trial Wizard, OncoTCap was required to model a basic description of the target domain of breast cancer. Using OncoTCap we described a model that included parameters for cell cycling time and turnover of the cell population, drug resistance pathways such as Multi Drug Resistance and bypassing of the certain genes leading to drug resistance and genetic pathways due to genetic mutations. The model also included a tumor location description with the breast being the primary tumor location and possible metastasis to the liver and the lung. OncoTCap also included a description of the trial drug, Pittamycin and the means to represent a regimen of Pittamycin applications and simulate their effect on the tumor growth. With an initial tumor cell count, a description of the tumor cell heterogeneity and a definition of the drug Pittamycin, OncoTCap performs a Monte Carlo simulation of breast cancer (see Day et al., 1998 for additional details). In this simulation, the tumor cells grow based on their cell kinetics parameters and the nature of their heterogeneous properties. The schedule and applications of the drug Pittamycin reduce the cell counts of the different types of tumor cells based on dose and the definition of the drug. OncoTCap provides two different ways of displaying the Monte Carlo simulation. In the Cancer Patient Simulator, the interactive simulation of tumor cell growth by means of a increasing graph for one single patient is shown. The relative cell counts of the various different celltypes determined by the heterogeneous properties are shown in different colors. The event window shows the various clinical and simulation events such as simulation start, diagnosis, metastasis, treatment and death. The Multiple Patient Simulator (MPS) runs the same Monte Carlo simulation as the Cancer Patient Simulator over multiple patients. Its display does not show interactive graphs. While the simulation is running, the MPS window shows a tally of the number of patients simulated, the number of responses, cures, death etc. At the end of the simulation, the MPS window also displays the event lists for each patient. A modified version of the MPS is used in the Trial Wizard. (see figure 1) Phase II Clinical Trial Wizard A Wizard in a computer program is a set of simple screens that walk the user through a complex cognitive task. The task is differentiated into multiple subtasks thus reducing the cognitive load. The task in this case was the design of a Phase II trial for study of the efficacy of the drug Pittamycin in the treatment of Metastatic Breast Cancer (i.e., breast cancer that has spread to other organs). Scaffolding using a wizard provides support to the user in two ways. First, the user is made aware of the expected components in a Phase II Clinical Trial by the contents of the various screens. Second, the user is able to accomplish the task by concentrating on one subtask at a time with less effort and confusion than if approaching the task as a whole. The screens of the wizard allow the user to easily navigate from one subtask to another and back again as needed. 1 This software is available for downloading at http:// www.pci.upmc.edu/tcap. Figure 1: Modified Multiple Patient Simulator summarizes the results of the simulation and allows students to examine individual patient event histories The computer supported clinical trial design and interpretation process can be divided into four functional components. The first component is the Introduction screen, which describes the objective of the wizard and the information the user will need to provide in the rest of the screens. The second component consists of a set of four well-defined steps that lead the user through the subtasks. In these screens, the user can input the various design parameters for the trial such as the Schedule (see figure 2), Dose Modifications due to toxicity (see figure 3 ), Off treatment criteria (see figure 4), and Statistical Trial design (see figure 5). The third component is the Summary, where the user's trial design from steps 14 are summarized in natural language and the user can click a button to run this simulation in the Multiple Patient Simulator (shown in figure 1). The modified MPS, which actually runs and displays the results of the clinical trial, forms the fourth component of the process. There is easy navigation between the various screens in the wizard via next and back buttons. The MPS has a back to the wizard button, which enables the user to view the results of the current trial and go back to the wizard to make further modifications to the design and re-run the trial. The final summary screen also provides the user with the ability to print out the description of the trial design with the date and time. The MPS allows the user to print out the results of any simulation run. The wizard was designed to provide the user with support for a typical phase II design. Complex designs involving more than one drug, complicated dose modifications and complex inclusion and off treatment criteria are outside the scope of this wizard. The wizard is not intended as a comprehensive trial design tool. The very process of simplifying and restricting the scope of possible designs to fit into a few wizard screens enhances the goal of making this tool easy to use. Figure 2: Step 1 of the Clinical Trial Design Wizard: Defining the dose and schedule Figure 4: Step 3 of the Clinical Trial Design Wizard: Deciding when individual patients will be taken offtreatment Figure 3: Step 2 of the Clinical Trial Design Wizard: Modifying the dose due occurrence of toxicity Figure 5: Step 4 of the Clinical Trial Design Wizard: Setting the statistical parameter Methods We tested the Phase II wizard with 24 fourth year medical students who were divided into 6 groups of 4 students each. Each group of students came into our lab for a 2 hour session and worked cooperatively at a computer. Prior to coming to the session, they were asked to develop a research proposal for a Phase II trial of the fictitious drug, Pittamycin. Students were given preclinical information and the results of the Phase I trial. The phase I trial information included the maximum dose that was tolerated by patients (MTD) and the types of toxicities that were observed (impairment of neurological function and formation of new blood cells). Each group of students had developed their initial research proposal prior to coming to the lab. The students then used the wizard to design and simulate their Phase II trials. The first author facilitated the group’s work with the wizard by 1) asking the students to summarize their initial proposal, 2) helping them with the any interface problems they might have had and making sure that they understood the relevant software features, 3) asking them to explain why they were making their changes, and 4) encouraging them to reflect on what they learned from this experience. Each run of the wizard generated a printout of the groups’ design. In addition, we printed out the results of the simulation that represented each group's final design. After using the software, students were asked to 1) evaluate the features of the software and 2) submit a revised research proposal (the “clinical concepts sheet”). In this paper, we describe the students’ behavior with the system, the improvement in their research proposals, and their evaluation of the software. Our data is drawn from four sources: 1) classroom observations, 2) printouts of design summaries for each simulation run as well as the printout of the results of their final design 3) group clinical concept sheets completed before and after using the wizard and 4) a questionnaire that asked about various aspects the software. The pre and post concepts sheets were coded for all the elements of the Phase II clinical design schema described earlier. Results and Discussion Using the simulation. The six student groups ran an average of 8.67 simulations during their sessions. The student's first simulation run was an implementation of their initial proposal. For the majority of the groups, their initial dosage and drug schedule was consistent with the MTD identified in from the Phase I trial information they received. In the first run through the trial wizard, when students reached the screen about dose modification, they were surprised to see it. In their pre-computer session concept sheet, students wrote about monitoring patients for toxicity, but they did not specify the actions to be taken if toxicities were observed. The students did not add these rules and remained consistent with their original design. They were less surprised when they had to consider the off-treatment criteria. Although poorly specified in their initial designs, the students appreciated the importance of setting these criteria. In the final step, the students needed to specify the operational characteristics of the clinical trial design. The program would then compute the number of patients needed and the number of responses that would allow them to conclude that the drug had some biological activity and was worthy of further study. The facilitator tried to elicit student understanding of these trial design specifications, but this remained a murky area for the students although they appreciated the need to specify those characteristics. Following this, the students simulated running the trial they had designed. The MPS summary included information about patient responses, deaths due to tumor burden, and deaths due to drug toxicity. The students closely watched the cumulating results. When students saw larger number of responses, they were initially perplexed as all of the patients still died, but on further consideration, the students realized that this was due to the fact that the patients selected for a Phase II trial usually had advanced disease. The students were able to view a survival plot that allowed them to see the increase in median survival time. This helped the students appreciate the therapeutic effect of the treatment. Their other concern was that along with any responses they observed, there were often many patients that died due to drug toxicity. Their perusal of individual patient records, particularly in the first few runs, led students to a deeper understanding of: 1. the types of toxicity that occurred 2 All sessions were audiotaped and videotaped and are being transcribed for further analysis. In this paper, we try to give a flavor, based on our observations, of what the sessions were like. 2. how long they took to resolve 3. why responses tended to occur fairly early in the treatment process. This concern with toxicity led all the groups to add dose modification rules in their second design. This screen was an important focus for student discussion about the nature of different types of toxicity. For example, students noted that neurologic toxicity caused very severe consequences (such as death of brain tissue) and only resolved slowly, whereas toxicity that impaired the body’s ability to make new blood cells resolved fairly quickly and was treatable. Thus, students tended to be conservative about dose modification for brain toxicity and so would severely reduce and more liberal in tolerating blood cell toxicity. Although students discussed the importance of only changing one variable at a time in order to understand the results of their experiments, this is rarely what they actually did. On the first few runs, the students changed 2-3 variables per run though this decreased over time. This may have occurred partly because the students were trying to manage multiple goals: maximizing responses while minimizing deaths due to toxicity. A strategy that several groups converged on involved using a very large dose of the drug early in the treatment and allowing time for the body systems to recover from the toxicity before giving another dose of the drug. This was effective because it killed the tumor cells before they developed resistance to the drug. The students were clearly engaged in the design and simulation process. All groups stayed for at least the two hour session and one of the groups even stayed for three. When asked to reflect on what they had learned, students often mentioned the statistics and an appreciation for the complexity of designing a clinical trial. Student assessment and evaluation of software. In the post-computer session clinical concept sheet, all the student groups modified their initial design to include specific dose modification rules, stop-treatment rules, as well as specifying their statistical parameters more completely. This suggests that students were constructing more elaborate trial design schemas. This elaboration is consistent with the scaffolding provided by the wizard. The conditional rules (dose modification and off-treatment criteria) and statistical parameters were either not fully specified or not considered in the initial trial designs but were included after the students were made aware of these factors in the wizard. The preand postresearch proposals were coded for the 23 components needed for a complete trial design. The mean score for the initial trial design was 13.67 (SD=4.27). The students improved on their postresearch proposals to a mean of 18.67 (SD=1.49; t(5)=2.87, p<0.05). The students rated the wizard positively, with a mean of 4.07 out of 5 on questions that asked students about different aspects of the learning environments’ ease of use, helpfulness for learning, and understandability. They noted the importance of having a facilitator and ranked this activity second only to their clinic time in value (students also received lectures and participated in a journal club). Their comments indicated that they liked receiving the rapid feedback on their experiments and having the opportunity to change variables and rerun their study. They did make useful suggestions about improving some of the printouts and screen displays. Conclusion Constructivist approaches to teaching and learning stress the importance of teaching scientific inquiry in the context of authentic problems that integrate both the inquiry process and subject matter learning (Collins, Brown, & Newman, 1989; Schauble et al., 1995). Working on authentic tasks, however can be extremely difficult for novices and requires additional support. This study demonstrates the effectiveness of scaffolding students’ scientific inquiry using a professional modeling tool. Using the clinical trial wizard, we help the students learn about the various components of the clinical trial design process. The dynamic feedback combined with ease of iteration proved crucial in allowing students to optimize their trial designs. Students went from a rudimentary Phase II trial schema to one that was more elaborated after working with the OncoTCap trial wizard. Our observations suggest that students used these tools not only to focus on the inquiry process, but to discuss important issues in cancer treatment as well. AcknowledgmentsWe thank Candace Johnson and Adam Brufsky for welcoming us into their course and helping with thedesign of this study. We thank Joseph Baar for help in designing the wizard text. This research is fundedby an R25 grant (1R25CA63548-01A2) from the National Cancer Institute. ReferencesBaker, L. M., & Dunbar, K. (1996). Constraints of the experimental design process in real-worldscience. In G. Cottrell (Ed.), Proceedings of the Eighteenth Annual Conference of the Cognitive ScienceSociety (pp. 154-159). Mahwah NJ: Erlbaum.Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craftsof reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essaysin honor of Robert Glaser (pp. 453-494). Hillsdale NJ: Erlbaum.Day, R., Shirey, W., Ramakrishnan, S., & Huang, Q. (1998, ). Tumor Biology ModelingWorkbench for Prospectively Evaluating Cancer Treatments. Paper presented at the 2nd IMACSInternational Multiconference : CESA'98 Computational Engineering in Systems Applications, Tunisia.Fuji, H., Marsh, C., Cairns, P., Sidransky, D., & Gabrielson, E. (1996). Genetic divergence in theclonal evolution of breast cancer. Cancer Research, 56, 1493-1497.Hmelo, C. E., & Guzdial, M. (1996). Of black and glass boxes: Scaffolding for learning anddoing. In D. C. Edelson & E. A. Domeshek (Eds.), Proceedings of ICLS 96 (pp. 128-134). CharlottesvilleVA: AACE.Nersessian, N. J. (1995). Should physicists preach what they practice?: Constructive modeling indoing and learning physics. Science and Education, 4(3).Ramakrishnan, S., Hmelo, C. E., Day, R. S., Shirey, W. E., & Huang, Q. (1998). TheIntegration of a Novice User Interface into A Professional Modeling Tool. Submitted to 1998 AMIAconference.Schauble, L., Glaser, R., Duschl, R., Schulze, S., & John, J. (1995). Students' understanding ofthe objectives and procedures of experimentation in the science classroom. Journal of the LearningSciences, 4, 131-166.Simon, R. (1993). Design and conduct of clinical trials. In V. T. DeVita, S. Hellman, & S. A.Rosenberg (Eds.), Cancer: Principles and Practice of Oncology (pp. 418-439). Philadelphia: Lippincott.Toth, E. (1996). Application of Computational modeling in scientific data visualization in highschools. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.